Updated File Upload to Auto Encode to UTF-8 #3438

elipe17 · 2025-01-21T22:22:59Z

Summary of Changes

Added new package to the frontend to better handle data file encoding errors
Updated FileUpload.jsx to use the new package to detect and subsequently transcode datafiles if they are not utf8 format

Pull request closes #2854

How to Test

List the steps to test the PR
These steps are generic, please adjust as necessary.

cd tdrs-frontend && docker-compose up --build
cd tdrs-backend && docker-compose up

Open http://localhost:3000/ and sign in.
Upload/submit a normal datafile to verify functionality
Take the same datafile open it in VS code. In the bottom right of VS code, you will see the encoding of the file. Click that and then select Save with Encoding. Choose a few encodings and submit each file as you change the encoding.
Verify that the file is uploaded, the backend doesn't report an invalid encoding, and ideally no data loss has occurred and the submission history reports the same output as the original submission from step 2. Some obvious new encodings that would break develop but not this PR are: UTF-8 with BOM, UTF-16 LE, and UTF-16 BE.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

files with UTF-8 with BOM encoding can be processed correctly
lfrohlich and/or adpennington confirmed that ACs are met.

Deliverable 2: Tested Code

Are all areas of code introduced in this PR meaningfully tested?
- If this PR introduces backend code changes, are they meaningfully tested?
- If this PR introduces frontend code changes, are they meaningfully tested?
Are code coverage minimums met?
- Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
- Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

Are backend code style checks passing on CircleCI?
Are frontend code style checks passing on CircleCI?
Are code maintainability principles being followed?

Deliverable 4: Accessible

Does this PR complete the epic?
Are links included to any other gov-approved PRs associated with epic?
Does PR include documentation for Raft's a11y review?
Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

Does this PR provide background for why coding decisions were made?
If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
If this PR introduces dependencies, are their licenses documented?
Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

Does the OWASP Scan pass on CircleCI?
Do manual code review and manual testing detect any new security issues?
If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

the purpose of the research
methods used to conduct the research
who participated in the research
what was tested and how
impact of research on TDP
(if applicable) final design mockups produced for TDP development

… always resolving with the loaded file

codecov · 2025-01-21T22:53:46Z

Codecov Report

Attention: Patch coverage is 76.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 91.37%. Comparing base (2269c21) to head (cf3f9c9).
Report is 8 commits behind head on develop.

Files with missing lines	Patch %	Lines
...-frontend/src/components/FileUpload/FileUpload.jsx	76.00%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3438   +/-   ##
========================================
  Coverage    91.37%   91.37%           
========================================
  Files          302      302           
  Lines         8689     8689           
  Branches       643      643           
========================================
  Hits          7940     7940           
  Misses         628      628           
  Partials       121      121

Flag	Coverage Δ
dev-backend	`91.26% <ø> (ø)`
dev-frontend	`92.22% <76.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tdrs-frontend/src/actions/reports.js	`100.00% <ø> (ø)`
...-frontend/src/components/FileUpload/FileUpload.jsx	`89.06% <76.00%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e66ad9a...cf3f9c9. Read the comment docs.

jtimpe

Working as expected, and i appreciate you cleaning up the functions a bit

raftmsohani

LGTM!

lhuxraft · 2025-02-03T18:38:12Z

@ADPennington

ADPennington · 2025-02-03T19:16:12Z

@elipe17 @lhuxraft pipeline failing on this branch.

…TANF-app into 2854-utf8-file-encode

elipe17 · 2025-02-03T20:37:28Z

@elipe17 @lhuxraft pipeline failing on this branch.

Even though all of the tests are passing, Jest is returning a status code of 1. My assumption is that this is an async/await issue in one of the tests. We get intermittent frontend test success because we get lucky with timing. I am still looking into where this could be happening.

ADPennington

confirmed that files with different encodings are processed consistently in TDP and error reports are also consistent. LGTM @elipe17 🚀

- Moved 2854-file-encoding-with-package over to new branch

f27da56

elipe17 self-assigned this Jan 21, 2025

elipe17 added 2 commits January 21, 2025 17:41

- Update onLoad to not generate more than one error state while still…

d29d61b

… always resolving with the loaded file

- Reject with an error object for consistency

1af1c37

elipe17 mentioned this pull request Jan 22, 2025

Catch File Encoding Errors with Package #3414

Closed

28 tasks

elipe17 requested review from andrew-jameson, raftmsohani and jtimpe January 22, 2025 13:51

elipe17 added frontend dev raft review This issue is ready for raft review labels Jan 22, 2025

jtimpe approved these changes Jan 22, 2025

View reviewed changes

Merge branch 'develop' into 2854-utf8-file-encode

a1d122a

lhuxraft added the a11y-review PR is ready for accessibility review label Jan 27, 2025

lhuxraft requested review from victoriaatraft and reitermb January 27, 2025 16:49

lhuxraft removed the a11y-review PR is ready for accessibility review label Jan 27, 2025

lhuxraft removed request for reitermb and victoriaatraft January 27, 2025 16:49

raftmsohani approved these changes Jan 28, 2025

View reviewed changes

jtimpe mentioned this pull request Jan 28, 2025

3398 - FRA Reports - search and upload #3445

Open

31 tasks

elipe17 added the QASP Review label Jan 29, 2025

lhuxraft removed raft review This issue is ready for raft review QASP Review labels Jan 29, 2025

elipe17 requested a review from ADPennington January 29, 2025 16:47

lhuxraft added the QASP Review label Jan 29, 2025

Merge branch 'develop' into 2854-utf8-file-encode

234bd70

lhuxraft mentioned this pull request Feb 3, 2025

3397 fra report types data file model #3412

Merged

32 tasks

elipe17 added 2 commits February 3, 2025 15:06

- Add waits for file change

d65f73b

Merge branch '2854-utf8-file-encode' of https://github.com/raft-tech/…

917304c

…TANF-app into 2854-utf8-file-encode

elipe17 and others added 5 commits February 3, 2025 16:08

- Adding waits in test

4225534

- UPdate dispatch count

a46dce0

- Add awaits in remaining tests

c74c917

Merge branch 'develop' into 2854-utf8-file-encode

d4d2ab7

Merge branch 'develop' into 2854-utf8-file-encode

2269c21

ADPennington added the Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI label Feb 6, 2025

ADPennington approved these changes Feb 7, 2025

View reviewed changes

ADPennington added Ready to Merge and removed QASP Review Deploy with CircleCI-qasp Deploy to https://tdp-frontend-qasp.app.cloud.gov through CircleCI labels Feb 7, 2025

Merge branch 'develop' into 2854-utf8-file-encode

cf3f9c9

elipe17 merged commit 3f8538f into develop Feb 7, 2025
12 checks passed

elipe17 deleted the 2854-utf8-file-encode branch February 7, 2025 16:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated File Upload to Auto Encode to UTF-8 #3438

Updated File Upload to Auto Encode to UTF-8 #3438

elipe17 commented Jan 21, 2025 •

edited by ADPennington

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

jtimpe left a comment

raftmsohani left a comment

lhuxraft commented Feb 3, 2025

ADPennington commented Feb 3, 2025

elipe17 commented Feb 3, 2025 •

edited

Loading

ADPennington left a comment

Updated File Upload to Auto Encode to UTF-8 #3438

Updated File Upload to Auto Encode to UTF-8 #3438

Conversation

elipe17 commented Jan 21, 2025 • edited by ADPennington Loading

Summary of Changes

How to Test

Deliverables

codecov bot commented Jan 21, 2025 • edited Loading

Codecov Report

jtimpe left a comment

Choose a reason for hiding this comment

raftmsohani left a comment

Choose a reason for hiding this comment

lhuxraft commented Feb 3, 2025

ADPennington commented Feb 3, 2025

elipe17 commented Feb 3, 2025 • edited Loading

ADPennington left a comment

Choose a reason for hiding this comment

elipe17 commented Jan 21, 2025 •

edited by ADPennington

Loading

codecov bot commented Jan 21, 2025 •

edited

Loading

elipe17 commented Feb 3, 2025 •

edited

Loading